1. Field of the Invention
The present invention relates to a method of and an apparatus for processing image, as well as to a medium which provided a program implementing the image processing method. More specifically, the present invention pertains to an image processing apparatus and method, as well as a medium, which permit easy retouching such as three-dimensional editorial modification on the image of a three-dimensional object that is contained in a two-dimensional displayed image.
2. Description of the Related Art
Various methods have been proposed that implement various processes on a two-dimensional image and extract information necessary for the processes from the two-dimensional image. Documents describing the methods include James D. Foley, Andries van Dam, Steven K. Feiner, and John F. Hughes, xe2x80x9cComputer Graphics, principles and practicexe2x80x9d, ADDISON-WESLEY PUBLISHING COMPANY, 1996 (hereinafter referred to as xe2x80x9cDocument 1xe2x80x9d), Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik, xe2x80x9cModeling and Rendering Architecture from Photographs: A hybrid geometry-and-image-based approachxe2x80x9d, proceedings of SIGGRAPH 96, pp. 11-20 (hereinafter referred to as xe2x80x9cDocument 2xe2x80x9d), Oliver Faugeras, xe2x80x9cThree-dimensional computer versionxe2x80x9d, The MIT press (hereinafter referred to as xe2x80x9cDocument 3xe2x80x9d), Kenneth P. Fishin, and Brian A. Barsky, xe2x80x9cFamily of New Algorithms for Soft Fillingxe2x80x9d, proceedings of SIGGRAPH 84, pp. 235-244 (hereinafter referred to as xe2x80x9cDocument 4xe2x80x9d), Pat Hanrahan and Paul Haeberli, xe2x80x9cDirect WYSIWYG Painting and Texuturing on 3D Shapesxe2x80x9d, proceedings of SIGGRAPH 90, pp. 215-233 (hereinafter referred to as xe2x80x9cDocument 5xe2x80x9d), Youichi Horry, Ken-ichi Anjyo, and Kiyoshi Arai, xe2x80x9cTour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Imagexe2x80x9d, proceedings of SIGGRAPH 97, pp. 225-232 (hereinafter referred to as xe2x80x9cDocument 6xe2x80x9d), and Michael Gleicher, xe2x80x9cImage Snappingxe2x80x9d, proceedings of SIGGRAPH 95, pp. 183-190 (hereinafter referred to as xe2x80x9cDocument 7xe2x80x9d).
In Document 1, image processing called xe2x80x9ctwo-dimensional paintxe2x80x9d is described in which a computer is used to perform processing in the same way as a designer or the like draws a picture on paper using a paintbrush or airbrushing (a technique that draws a picture by spraying paints on paper).
In this type of conventional two-dimensional paint, even when a three-dimensional object is displayed in an image, the image itself is treated on a two-dimensional plane. Accordingly, when characters are rendered irrespective of the direction of the three-dimensional object in a three-dimensional space which is displayed on the two-dimensional plane, or a figure is added, the image looks unnatural.
For example, in the case where a house-shaped three-dimensional object, as shown in FIG. 1A, is displayed, and characters are rendered on walls of the house without the direction of the characters, the characters do not look as if they are written on the walls, as shown in FIG. 1B. For adding a parallelepiped room to a wall of the house, in the case where a rectangle is rendered on the wall without the direction of the rectangle, the image looks unnatural, as shown in FIG. 1C. In the case where a cylinder is displayed in a two-dimensional image as shown in FIG. 2A, when characters are rendered on the side surface without ignoring the curvature of the side surface, the characters do not look as if they are written on the side surface, as shown in FIG. 2B.
Accordingly, in order that an image may not look unnatural in two-dimensional paint, it is required that a character or figure be rendered being transformed so as to match the direction of a three-dimensional object displayed in the two-dimensional image. Performing operations for the rendering requires a degree of experience.
Therefore, there is a method in which a user uses a ten-key pad or graphical user interface (GUI) to input an angle of inclination of the three-dimensional object displayed on the two-dimensional image so that a computer uses the input to transform a new character or figure to be rendered. In this method, the user must adjust the angle of inclination of the three-dimensional object to be input to the computer while viewing the rendered character or figure so that the rendered character or figure does not look unnatural. The adjustment also requires a degree of experience.
As described above, when the user instructs the rendering of a character or figure ignoring the direction of a three-dimensional object displayed on a two-dimensional image, the computer cannot display the character or figure as it looks natural, in other words, an image looking as if the character or figure was originally positioned cannot be obtained. This is due to lack of information on the position of the three-dimensional object displayed in the two-dimensional image and information (a position at which a landscape or the like was observed in the case where the two-dimensional image was obtained by performing image capture on a picture) on the position of image capture for the two-dimensional image.
Accordingly, there is a method that uses computer vision to find, from a two-dimensional image, the position of a three-dimensional object in a three-dimensional space displayed in the two-dimensional image, and the image capture position of the three-dimensional object.
In other words, in Document 2, a method is disclosed in which a three-dimensional shape such as a parallelepiped is correlated using a GUI with a structure in a plurality of photographs, and the size of the structure and the photographing position are found. In Document 3, other various methods for finding the position of a three-dimensional object and an image capture position on the object are disclosed.
In the above-described methods using the computer vision to compute a three-dimensional object position and an image capture position on the object, the principles of triangulation are used. This requires a plurality of images obtained by performing image capture on the same three-dimensional object from plurality of image capture positions. However, when two-dimensional paint is performed, the images are not always prepared, and when the two-dimensional images are photographs of a picture, the plurality of images is not used in general.
If the plurality of images obtained by performing image capture on the same three-dimensional object from the plurality of image capture positions can be prepared, the corresponding positions (e.g., vertices of the roof, etc., of a structure as a three-dimensional object) of the same three-dimensional object displayed on the images must be designated in order for the computer to compute the position of the three-dimensional object and the image capture position. The operation of designation for the images is complicated, and requires a time. In addition, in the case where based on the position (in the three-dimensional space) of the three-dimensional object displayed in the two-dimensional image and the image capture position, three-dimensionally natural rendering is performed, three-dimensional data, such as three-dimensional coordinates generated when the three-dimensional object is viewed from the image capture position, must be processed, which requires a great amount of operations for the processing.
In Document 4, a method for color processing for two-dimensional paint is disclosed.
In two-dimensional paint, the user uses a GUI to select a color for rendering and to perform rendering using the selected color. The color and brightness of a three-dimensional object displayed in the two-dimensional image vary depending on the positional relationship between the direction of the object surface and illumination. For example, painting an entire surface of the three-dimensional object in the same color (the same RGB levels) causes an unnatural rendering result. Accordingly, for obtaining a natural rendering result, a painting color must be gradationally changed considering the positional relationship between the direction of a surface to be painted and illumination. In particular, in the case where a surface to be painted is curved, pixel levels needs to be sequentially changed so that the rendering result is natural. Thus, a painting color must be selected for, for example, several pixels, which requires a great amount of operations.
Accordingly, in Document 4, a method called xe2x80x9ctint fillxe2x80x9d is disclosed in which in the case where a pixel color is a mixture of two colors and the two colors are known, either color can be changed into another color. The use of tint fill can efficiently cope with an area in which a foreground and a background are mixed, but cannot cope with object shading and light reflected by the object.
In Document 5, a method that transforms and pastes a character or handwritten figure input by operating a tablet or mouse along the surface of a three-dimensional object displayed in a two-dimensional image is disclosed.
Since the method disclosed in Document 5 uses three-dimensional information (e.g., the position of the three-dimensional object in a three-dimensional space, an image capture position, size, inclination, etc.) on the three-dimensional object displayed in the two-dimensional image, the method in Document 5 needs to recognize the three-dimensional information by using computer vision. However, when computer vision is used to recognize the three-dimensional information, a plurality of images are required as described above, and an amount of operations greatly increases.
Although the method in Document 5 can also use a three-dimensional model of a three-dimensional object generated in the computer, it must process three-dimensional data and requires a great amount of operations.
Regarding three-dimensional model generating, Document 2 discloses a technique that includes the steps of using computer vision technology to extract an outline of the shape of a three-dimensional object displayed in the two-dimensional image from the two-dimensional image, pasting a captured image onto the shape surface (performing texture mapping), and using these steps to generate a two-dimensional image viewed from a free viewpoint (captured from a free image capture position). Similar to the above-described cases, this technique requires a plurality of images because it uses computer vision technology to extract, from a two-dimensional image, an outline of the shape of a three-dimensional object displayed in the two-dimensional image.
In Document 6, a method is disclosed in which based on a single two-dimensional image, an image viewed from a viewpoint excluding the image capture position of the two-dimensional image is generated. This method only generates an image viewed from a viewpoint excluding the image capture position, and it is based on the assumption that a three-dimensional object displayed in a two-dimensional image or another adjacent object has a very simple shape of a plate. Accordingly, almost no serious problem occurs when an image viewed from another viewpoint is approximately generated. However, when a character or figure is added to the two-dimensional image, it is impossible to make the rendering result natural. For example, in the case where a cylinder is displayed in a two-dimensional image, a natural image formed by adding a character or figure on the side surface of the cylinder cannot be obtained.
In addition, it is required in many cases that when a two-dimensional image is processed, the user designates predetermined characteristic points (e.g., vertices) of a three-dimensional object in the two-dimensional image. The characteristic points are designated by operating a mouse or input pen, but it is difficult to accurately designate vertices of the three-dimensional object in the two-dimensional image. Accordingly, Document 7 discloses a method that includes the steps of extracting an edge (a portion whose brightness and color greatly change) in the two-dimensional image, and moving a cursor on the edge when the cursor is in the vicinity of the edge. Since all edges in the two-dimensional image are detected in the method in Document 7, sensitive response to noise or a texture in the two-dimensional image occurs, and the cursor may be moved to an unnecessary position.
Although many image processing techniques have been proposed heretofore as described above, these techniques require certain levels of skill and experience, as well as time, or necessitates a plurality of images of an object as viewed in different directions. These techniques therefore cannot suitably be used for the purpose of effecting three-dimensional edition or other retouching on the images of characters and patterns which are contained in a two-dimensional displayed image.
Thus, no image processing technique has heretofore been proposed which would enable easy three-dimensional edition or retouching on characters or patterns in a two-dimensional displayed image.
Accordingly, it is a primary object of the present invention to provide an image processing technique which permits a user to easily effect three-dimensional editorial work or other retouching on a two-dimensional displayed image.
To this end, according to one aspect of the present invention, there is provided an image processing apparatus for processing a two-dimensional moving picture, comprising: operating means operable to enable designation of shape information concerning the shape of a surface that forms part of a three-dimensional object displayed in the moving picture and designation of a characteristic point which is located on the moving picture; computing means for determining, based on the shape information and the characteristic point, a correlation equation expressing the correlation between a projected image which is obtained by projecting the surface constituting part of the three-dimensional object and an expanded image which is obtained by developing the surface into two-dimensional surface; first transforming means for transforming, based on the correlation equation, the projected image of the surface constituting part of the three-dimensional object in the moving picture into the expanded image; storing means for storing the expanded image; retouching means for effecting retouching on the expanded image stored in the storing means; and second transforming means for transforming, based on the correlation equation, the retouched expanded image back to the projected image in the moving picture.
According to another aspect of the present invention, there is provided an image processing method for processing a two-dimensional moving picture, comprising the steps of: preparing operating means operable to enable designation of shape information concerning the shape of a surface that forms part of a three-dimensional object displayed in the moving picture and designation of a characteristic point which is located on the moving picture; computing, in response to the designation of the shape information and the characteristic point by the operating means, a correlation equation expressing the correlation between a projected image which is obtained by projecting the surface constituting part of the three-dimensional object and an expanded image which is obtained by developing the surface into two-dimensional surface, based on the shape information and the characteristic point; executing first transformation for transforming, based on the correlation equation, the projected image of the surface constituting part of the three-dimensional object in the moving picture into the expanded image; storing the expanded image; retouching the expanded image stored in the storing step; and executing a second transformation for transforming, based on the correlation equation, the retouched expanded image back to the projected image in the moving picture.
According to still another aspect of the present invention, there is provided a medium for enabling a computer to execute a computer program for conducting image processing on a two-dimensional moving picture, the program comprising the steps of: computing, in response to designation of shape information concerning the shape of a surface that forms part of a three-dimensional object displayed in the moving picture and designation of a characteristic point which is located on the moving picture, the designation being made through operation of operating means that enables designation of the shape information and the characteristic point, a correlation equation expressing the correlation between a projected image which is obtained by projecting the surface constituting part of the three-dimensional object and an expanded image which is obtained by developing the surface into two-dimensional surface, based on the shape information and the characteristic point; executing first transformation for transforming, based on the correlation equation, the projected image of the surface constituting part of the three-dimensional object in the moving picture into the expanded image; storing the expanded image; retouching the expanded image stored in the storing step; and executing a second transformation for transforming, based on the correlation equation, the retouched expanded image back to the projected image in the moving picture.
The above and other objects, features and advantages of the present invention will become clear from the following description of the preferred embodiment when the same is read in conjunction with the accompanying drawings.